System and method of guiding handwriting diagram input

ABSTRACT

A system, method and computer program product for guiding hand-drawing of diagrams including text and non-text elements on a computing device are provided. The computing device has a processor and a non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor. Display is performed, on an interactive display of the computing device, of a guide element associated with at least one diagram element of displayed handwriting diagram input. The guide element is configured with a depiction of the at least one diagram element in recognized form.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to European Application No. 15290270.6filed on Oct. 19, 2015, the entire contents of which is incorporated byreference herein.

TECHNICAL FIELD

The present invention relates generally to the field of computing deviceinterfaces capable of recognizing user input handwriting of variousgraphics and text. In particular, the present invention provides systemsand methods for guiding the hand-drawn input of diagram elements toproduce digital diagram documents and interaction therewith.

BACKGROUND

Computing devices continue to become more ubiquitous to daily life. Theytake the form of computer desktops, laptop computers, tablet computers,e-book readers, mobile phones, smartphones, wearable computers, globalpositioning system (GPS) units, enterprise digital assistants (EDAs),personal digital assistants (PDAs), game consoles, and the like.Further, computing devices are being incorporated into vehicles andequipment, such as cars, trucks, farm equipment, manufacturingequipment, building environment control (e.g., lighting, HVAC), and homeand commercial appliances.

Computing devices generally consist of at least one processing element,such as a central processing unit (CPU), some form of memory, and inputand output devices. The variety of computing devices and theirsubsequent uses necessitate a variety of interfaces and input devices.One such input device is a touch sensitive surface such as a touchscreen or touch pad wherein user input is received through contactbetween the user's finger or an instrument such as a pen or stylus andthe touch sensitive surface. Another input device is an input surfacethat senses gestures made by a user above the input surface. A furtherinput device is a position detection system which detects the relativeposition of either touch or non-touch interactions with a non-touchsurface. Any of these methods of input can be used generally for thehandwritten or hand-drawn input of drawings and text which input isinterpreted using a handwriting recognition system or method.

One application of handwriting recognition in computing devices is inthe creation of diagrams which are hand-drawn on a computing device tobe converted into typeset versions. Diagrams are drawings that explainor show arrangement and relations (as of parts). Diagrams generallyinclude shapes having arbitrary or specific meanings and text withrelationships to these shapes. There are many type of diagrams, such asflowcharts, organizational charts, concept maps, spider maps,block/architecture diagrams, mind-maps, block diagrams, Venn diagramsand pyramids. Diagrams generally include shapes, defining diagram blocksor containers, of different type (e.g., ovals, rectangles, diamonds,nested relationships of containers within containers), and connectors,of different type (e.g., straight lines, curved lines, straight arrows,curved arrows, branched lines), which connect or designate relationshipsbetween the diagram blocks, and text, contained in containers,associated to connectors or defined in text blocks having no associatedshape or container. These myriad possible variations of combining thebase components of shapes (connections with or without containers) andtext in diagrams can cause issues for the accurate recognition of theseelements input as hand-drawn or written content to a computing device.

Diagrams are particularly used in education and business settings wherethe user of the computing device creates a diagram, for example, duringa lecture or meeting to capture concepts, issues or solutions beingdiscussed. This is usually done by the user launching a handwrittendiagram or sketch application on the computing device which accepts andinterprets, either locally in the device or remotely via acommunications link of the device, hand-drawn input on a touch sensitivesurface or a surface monitored by a relative position detection system.

Conventionally such handwritten diagramming applications are limited intheir capabilities to handle the above-described complexity ofdiagramming and typically constrain users to adopt behaviors or acceptcompromises which do not reflect the user's original intent. As a resultsome conventional handwritten diagramming applications force users tonavigate menus to select and draw shapes and insert text in relation toshapes. As such, users are unable to draw shapes and connectorsnaturally or freely. Some other conventional handwritten diagrammingapplications rely on the order in which users draw different strokes tothereby guide the interpretation for recognition as expected behavior isfollowed. For example, the user may need to first draw two blocks/boxesbefore being able to define a connector between those boxes, or may haveto draw a box before adding text thereto. This however is difficult forusers, as they need to learn and implement the drawing/writing ordersrequired which may need to be re-learnt if the application is not oftenused, and is non-intuitive, such that the ability to quickly capturediagrams is not supported. For example, a user may wish to preparepresentations on the go with a portable computing device, such as atablet, or a user may wish to jot down a flow chart that their teacherhas drawn in class on a computing device, such as a laptop with atouchscreen, and as such users need to be able to draw clear diagramswith mixed content without being an expert of the dedicated, cumbersomesoftware.

Making the handwritten diagramming application smarter helps supportusers. That is, the application may be able to distinguish betweendifferent shapes, such as between blocks and connectors, and betweenshapes and text, thereby providing users with more freedom when creatingdiagrams. Even in conventional applications in which hand-drawn shapesand handwritten text are recognized well with reasonable creativefreedom offered to users, typically the ability to change the drawndiagrams, such as to edit elements of the diagram to add, omit orreplace elements, is limited where only certain operations are availableand only available on the typeset version of the diagram, not on thehandwritten input, so-called digital ink, and/or requires gestures to belearnt or selection to be made via menus, as described above.

Users typically desire real-time feedback, which is during writing, ofthe recognition of shapes and text when using diagramming applications.Some conventional applications provide such feedback mechanisms to usersbut these are generally limited in their effectiveness or are cumbersomein design. For example, available diagramming applications providefeedback by typesetting the handwritten input automatically (e.g.,on-the-fly). However, such systems generally distract users from theflow of input since they must wait for the typesetting to be performedbefore carrying on with drawing. Other available handwritten note takingapplications provide feedback by listing recognition candidates and thelike during input. This is also very distracting for users.

SUMMARY

The examples of the present invention that are described herein belowprovide systems, methods and a computer program product for use indiagram creation with handwriting input to a computing device. Thecomputer program product has a non-transitory computer readable mediumwith a computer readable program code embodied therein adapted to beexecuted to implement the method.

The computing device is connected to an input device in the form of aninput surface. A user is able to provide input by applying pressure toor gesturing above the input surface using either his or her finger oran instrument such as a stylus or pen. The present system and methodmonitors the input strokes.

The computing device has a processor and at least one application fordetecting and recognizing the handwriting input under control of theprocessor. The at least one system application is configured to causedisplay of, on an interactive display of the computing device, a guideelement associated with at least one diagram element of displayedhandwriting diagram input, the guide element configured with a depictionof the at least one diagram element in recognized form.

Another aspect of the disclosed system and method provides the guideelement as including a dynamic prompter which dynamically displays theat least one diagram element depiction as the handwriting is input. Thedynamic prompter includes at least one interactive prompter element, theat least one non-transitory computer readable medium configured todisplay of an action menu in response to receiving interaction with theat least one interactive prompter element.

Another aspect of the disclosed system and method provides display ofthe depiction of a non-text element in icon form. The non-text elementmay be a shape element of the diagram, with the icon depicting theshape.

Another aspect of the disclosed system and method provides display ofthe depiction of a text element in typeset ink.

BRIEF DESCRIPTION OF THE DRAWINGS

The present system and method will be more fully understood from thefollowing detailed description of the examples thereof, taken togetherwith the drawings. In the drawings like reference numerals depict likeelements. In the drawings:

FIG. 1 shows a block diagram of a computing device in accordance with anexample of the present system and method;

FIG. 2 shows a block diagram of a system for handwriting recognition inaccordance with an example of the present system and method;

FIG. 3 shows a block diagram illustrating detail of the handwritingrecognition system of FIG. 2 in accordance with an example of thepresent system and method;

FIG. 4 shows an example hand-drawn diagram depicting a guide element;

FIG. 5 shows input of a first hand-drawn text element of the examplediagram with associated components of the guide element;

FIG. 6 shows subsequent input of a hand-drawn shape element of theexample diagram with associated components of the guide element;

FIG. 7 shows the diagram state of FIG. 6 depicting selection of the textelement with associated components of the guide element;

FIG. 8 shows subsequent input of a hand-drawn shape element of theexample diagram associated with the text element of FIG. 7 withassociated components of the guide element;

FIG. 9 shows subsequent input of a hand-drawn shape element of theexample diagram with associated components of the guide element;

FIG. 10 shows alternative subsequent input (to the input of FIG. 9) of ahand-drawn element of the example diagram with associated components ofthe guide element;

FIG. 11 shows the diagram state of FIG. 10 depicting menu display fromselection of a component of the guide element for the element;

FIG. 12 shows the diagram state of FIG. 10 depicting another menudisplay from selection of another component of the guide element for theelement;

FIG. 13 shows the diagram state of FIG. 9 depicting selection of thecombined shape element input of FIGS. 8 and 9 with associated componentsof the guide element;

FIG. 14 shows the diagram state of FIG. 13 depicting menu display fromselection of a component of the guide element for the combined shapeelements;

FIG. 15 shows the diagram state of FIG. 13 depicting another menudisplay from selection of another component of the guide element for thecombined shape elements;

FIG. 16 shows input of a further hand-drawn text element of the examplediagram with associated components of the guide element;

FIG. 17 shows alternative input (to the input of FIG. 16) of furtherhand-drawn shape and text elements of the example diagram withassociated components of the guide element;

FIG. 18 shows the diagram state of FIG. 17 depicting menu display fromselection of a component of the guide element for the shape element;

FIG. 19 shows subsequent input of a hand-drawn shape element of theexample diagram with associated components of the guide element;

FIG. 20 shows the diagram state of FIG. 19 depicting selection of theshape and text element inputs of FIGS. 16 and 19 with associatedcomponents of the guide element;

FIG. 21A shows the diagram state of FIG. 16 depicting selection of thetext element input of FIG. 16 with associated components of the guideelement;

FIG. 21B shows alternative input (to the inputs of FIGS. 16 and 19) of ahand-drawn shape element of the example diagram depicting selection ofthat shape element with associated components of the guide element;

FIG. 22 shows the example diagram displayed in typeset ink;

FIG. 23 shows the diagram state of FIG. 22 depicting selection of theshape and text element inputs of FIGS. 16 and 19 with associatedcomponents of the guide element;

FIG. 24 shows the diagram state of FIG. 23 depicting menu display fromselection of a component of the guide element for the selected elements;

FIG. 25 shows the diagram state of FIG. 23 depicting another menudisplay from selection of another component of the guide element for theselected elements;

FIG. 26 shows the diagram state of FIG. 23 depicting another menudisplay from selection of another component of the guide element for theselected elements;

FIG. 27 shows the diagram state of FIG. 22 depicting multi-selection ofthe shape and text elements with associated components of the guideelement;

FIG. 28 shows the diagram state of FIG. 27 depicting menu display fromselection of a component of the guide element for the multi-selectedelements;

FIG. 29 shows further input of a hand-drawn free-form shape element tothe example diagram of FIG. 22 with associated components of the guideelement;

FIG. 30 shows the diagram state of FIG. 29 depicting selection of thefree-form shape with associated components of the guide element; and

FIG. 31 shows the diagram state of FIG. 30 depicting menu display fromselection of a component of the guide element for the selected free-fromelement.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant teachings. However, it should be apparent to those skilledin the art that the present teachings may be practiced without suchdetails. In other instances, well known methods, procedures, components,and/or circuitry have been described at a relatively high-level, withoutdetail, in order to avoid unnecessarily obscuring aspects of the presentteachings.

Reference to and discussion of directional features such as up, down,above, below, lowest, highest, horizontal, vertical, etc., are made withrespect to the Cartesian coordinate system as applied to the inputsurface on which the input to be recognized is made. Further, terms suchas left and right are made in relation to the reader's frame ofreference when viewing the drawings. Furthermore, the use of the term‘text’ in the present description is understood as encompassing allalphanumeric characters, and strings thereof, in any written languageand common place non-alphanumeric characters, e.g., symbols, used inwritten text. Further still, the term ‘non-text’ in the presentdescription is understood as encompassing freeform handwritten orhand-drawn content and rendered text and image data, as well asnon-alphanumeric characters, and strings thereof, and alphanumericcharacters, and strings thereof, which are used in non-text contexts.Furthermore, the examples shown in these drawings are in a left-to-rightwritten language context, and therefore any reference to positions canbe adapted for written languages having different directional formats.

The various technologies described herein generally relate to capture,processing and management of hand-drawn and handwritten content onportable and non-portable computing devices in a manner which retainsthe inputted style of the content while allowing conversion to afaithful typeset or beautified version of that content. The systems andmethods described herein may utilize recognition of users' naturalwriting and drawing styles input to a computing device via an inputsurface, such as a touch sensitive screen, connected to, or of, thecomputing device or via an input device, such as a digital pen or mouse,connected to the computing device or via a surface monitored by aposition detection system. Whilst the various examples are describedwith respect to recognition of handwriting input using so-called onlinerecognition techniques, it is understood that application is possible toother forms of input for recognition, such as offline recognition inwhich images rather than digital ink are recognized. The termshand-drawing and handwriting are used interchangeably herein to definethe creation of digital content by users through use of their handseither directly onto a digital or digitally connected medium or via aninput tool, such as a hand-held stylus. The term “hand” is used hereinto provide concise description of the input techniques, however the useof other parts of a users' body for similar input is included in thisdefinition, such as foot, mouth and eye.

FIG. 1 shows a block diagram of an example computing device 100. Thecomputing device may be a computer desktop, laptop computer, tabletcomputer, e-book reader, mobile phone, smartphone, wearable computer,digital watch, interactive whiteboard, global positioning system (GPS)unit, enterprise digital assistant (EDA), personal digital assistant(PDA), game console, or the like. The computing device 100 includescomponents of at least one processing element, some form of memory andinput and/or output (I/O) devices. The components communicate with eachother through inputs and outputs, such as connectors, lines, buses,cables, buffers, electromagnetic links, networks, modems, transducers,IR ports, antennas, or others known to those of ordinary skill in theart.

The computing device 100 has at least one display 102 for outputtingdata from the computing device such as images, text, and video. Thedisplay 102 may use LCD, plasma, LED, iOLED, CRT, or any otherappropriate technology that is or is not touch sensitive as known tothose of ordinary skill in the art. The display 102 may be co-locatedwith at least one input surface 104 or remotely connected thereto. Theinput surface 104 may employ technology such as resistive, surfaceacoustic wave, capacitive, infrared grid, infrared acrylic projection,optical imaging, dispersive signal technology, acoustic pulserecognition, or any other appropriate technology as known to those ofordinary skill in the art to receive user input in the form of a touch-or proximity-sensitive surface. The input surface 104 may be bounded bya permanent or video-generated border that clearly identifies itsboundaries. The input surface 104 may a non-touch sensitive surfacewhich is monitored by a position detection system.

In addition to the input surface 104, the computing device 100 mayinclude one or more additional I/O devices (or peripherals) that arecommunicatively coupled via a local interface. The additional I/Odevices may include input devices such as a keyboard, mouse, scanner,microphone, touchpads, bar code readers, laser readers, radio-frequencydevice readers, or any other appropriate technology known to those ofordinary skill in the art. Further, the I/O devices may include outputdevices such as a printer, bar code printers, or any other appropriatetechnology known to those of ordinary skill in the art. Furthermore, theI/O devices may include communications devices that communicate bothinputs and outputs such as a modulator/demodulator (modem; for accessinganother device, system, or network), a radio frequency (RF) or othertransceiver, a telephonic interface, a bridge, a router, or any otherappropriate technology known to those of ordinary skill in the art. Thelocal interface may have additional elements to enable communications,such as controllers, buffers (caches), drivers, repeaters, andreceivers, which are omitted for simplicity but known to those of skillin the art. Further, the local interface may include address, control,and/or data connections to enable appropriate communications among theother computer components.

The computing device 100 also includes a processor 106, which is ahardware device for executing software, particularly software stored inthe memory 108. The processor can be any custom made or commerciallyavailable general purpose processor, a central processing unit (CPU), asemiconductor based microprocessor (in the form of a microchip orchipset), a macroprocessor, microcontroller, digital signal processor(DSP), application specific integrated circuit (ASIC), fieldprogrammable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, statemachine, or any combination thereof designed for executing softwareinstructions known to those of ordinary skill in the art. Examples ofsuitable commercially available microprocessors are as follows: aPA-RISC series microprocessor from Hewlett-Packard Company, an 80×86 orPentium series microprocessor from Intel Corporation, a PowerPCmicroprocessor from IBM, a Sparc microprocessor from Sun Microsystems,Inc., a 68xxx series microprocessor from Motorola Corporation, DSPmicroprocessors, or ARM microprocessors.

The memory 108 may include any one or a combination of volatile memoryelements (e.g., random access memory (RAM, such as DRAM, SRAM, orSDRAM)) and nonvolatile memory elements (e.g., ROM, EPROM, flash PROM,EEPROM, hard drive, magnetic or optical tape, memory registers, CD-ROM,WORM, DVD, redundant array of inexpensive disks (RAID), another directaccess storage device (DASD)). Moreover, the memory 108 may incorporateelectronic, magnetic, optical, and/or other types of storage media. Thememory 108 can have a distributed architecture where various componentsare situated remote from one another but can also be accessed by theprocessor 106. Further, the memory 108 may be remote from the device,such as at a server or cloud-based system, which is remotely accessibleby the computing device 100. The memory 108 is coupled to the processor106, so the processor 106 can read information from and writeinformation to the memory 108. In the alternative, the memory 108 may beintegral to the processor 106. In another example, the processor 106 andthe memory 108 may both reside in a single ASIC or other integratedcircuit.

The software in the memory 108 includes an operating system 110 and anapplication 112. The software optionally further includes a handwritingrecognition (HWR) system 114 which may each include one or more separatecomputer programs. Each of these has an ordered listing of executableinstructions for implementing logical functions. The operating system110 controls the execution of the application 112 (and the HWR system114). The operating system 110 may be any proprietary operating systemor a commercially available operating system, such as WEBOS, WINDOWS®,MAC and IPHONE OS®, LINUX, and ANDROID. It is understood that otheroperating systems may also be utilized.

The application 112 includes one or more processing elements related todetection, management and treatment of hand-drawn shapes and handwrittentext input by users (discussed in detail later). The software may alsoinclude one or more other applications related to handwritingrecognition, different functions, or both. Some examples of otherapplications include a text editor, telephone dialer, contactsdirectory, instant messaging facility, computer-aided design (CAD)program, email program, word processing program, web browser, andcamera. The application 112, and the other applications, includeprogram(s) provided with the computing device 100 upon manufacture andmay further include programs uploaded or downloaded into the computingdevice 100 after manufacture.

The present system and method make use of the HWR system 114 in order torecognize handwritten input to the device 100, including handwrittentext and hand-drawn shapes, e.g., non-text. The HWR system 114, withsupport and compliance capabilities, may be a source program, executableprogram (object code), script, application, or any other entity having aset of instructions to be performed. When a source program, the programneeds to be translated via a compiler, assembler, interpreter, or thelike, which may or may not be included within the memory, so as tooperate properly in connection with the operating system. Furthermore,the handwriting recognition system with support and compliancecapabilities can be written as (a) an object oriented programminglanguage, which has classes of data and methods; (b) a procedureprogramming language, which has routines, subroutines, and/or functions,for example but not limited to C, C++, Pascal, Basic, Fortran, Cobol,Perl, Java, Objective C, Swift, and Ada; or (c) functional programminglanguages for example but no limited to Hope, Rex, Common Lisp, Scheme,Clojure, Racket, Erlang, OCaml, Haskell, Prolog, and F#. Alternatively,the HWR system 114 may be a method or system for communication with ahandwriting recognition system remote from the device, such as server orcloud-based system, but is remotely accessible by the computing device100 through communications links using the afore-mentionedcommunications I/O devices of the computing device 100. Further, theapplication 112 and the HWR system 114 may operate together accessinginformation processed and stored in the memory 108, for example, by eachsystem, or be combined as a single application.

Strokes entered on or via the input surface 104 are processed by theprocessor 106 as digital ink. A user may enter a stroke with a finger orsome instrument such as a pen or stylus suitable for use with the inputsurface. The user may also enter a stroke by making a gesture above theinput surface 104 if technology that senses motions in the vicinity ofthe input surface 104 is being used, or with a peripheral device of thecomputing device 100, such as a mouse or joystick. A stroke ischaracterized by at least the stroke initiation location, the stroketermination location, and the path connecting the stroke initiation andtermination locations. Because different users may naturally write thesame object, e.g., a letter, a shape, a symbol, with slight variations,the HWR system accommodates a variety of ways in which each object maybe entered whilst being recognized as the correct or intended object.

FIG. 2 is a schematic pictorial of an example of the HWR system 114, ineither its local (i.e., loaded on the device 100) or remote (i.e.,remotely accessible by the device 100) forms. The HWR system 114includes stages such as preprocessing 116, recognition 118 and output120. The preprocessing stage 116 processes the digital ink to achievegreater accuracy and reducing processing time during the recognitionstage 118. This preprocessing may include normalizing of the pathconnecting the stroke initiation and termination locations by applyingsize normalization and/or methods such as B-spline approximation tosmooth the input. The preprocessed strokes are then passed to therecognition stage 118 which processes the strokes to recognize theobjects formed thereby. The recognized objects are then output 120 tothe memory 108 and the display 102 as a digital ink or typeset inkversions of the handwritten elements/characters and hand-drawn shapes.

The recognition stage 118 may include different processing elements orexperts. FIG. 3 is a schematic pictorial of the example of FIG. 2showing schematic detail of the recognition stage 118. Three experts, asegmentation expert 122, a recognition expert 124, and a language expert126, are illustrated which collaborate through dynamic programming togenerate the output 120.

The segmentation expert 122 defines the different ways to segment theinput strokes into individual element hypotheses, e.g., alphanumericcharacters and mathematical operators, text characters, individualshapes, or sub expression, in order to form expressions, e.g., words,mathematical equations, or groups of shapes. For example, thesegmentation expert 122 may form the element hypotheses by groupingconsecutive strokes of the original input to obtain a segmentation graphwhere each node corresponds to at least one element hypothesis and whereadjacency constraints between elements are handled by the nodeconnections. Alternatively, the segmentation expert 122 may employseparate experts for different input types, such as text, drawings,equations, and music notation.

The recognition expert 124 provides classification of the featuresextracted by a classifier 128 and outputs a list of element candidateswith probabilities or recognition scores for each node of thesegmentation graph. Many types of classifiers exist that could be usedto address this recognition task, e.g., Support Vector Machines, HiddenMarkov Models, or Neural Networks such as Multilayer Perceptrons, Deep,Convolutional or Recurrent Neural Networks. The choice depends on thecomplexity, accuracy, and speed desired for the task.

The language expert 126 generates linguistic meaning for the differentpaths in the segmentation graph using language models (e.g., grammar orsemantics). The expert 126 checks the candidates suggested by the otherexperts according to linguistic information 130. The linguisticinformation 130 can include a lexicon(s), regular expressions, etc. Thelanguage expert 126 aims at finding the best recognition path. In oneexample, the language expert 126 does this by exploring a language modelsuch as final state automaton (determinist FSA) representing the contentof linguistic information 130. In addition to the lexicon constraint,the language expert 126 may use statistical information modeling for howfrequent a given sequence of elements appears in the specified languageor is used by a specific user to evaluate the linguistic likelihood ofthe interpretation of a given path of the segmentation graph.

The application 112 provided by the present system and method allowsusers, such as students, academic and working professionals, to createhandwritten diagrams and have those diagrams faithfully recognized usingthe HWR system 114 independent of the type of diagram created, e.g.,flowcharts, organizational charts, concept maps, spider maps,block/architecture diagrams, mind-maps, block diagrams, Venn diagramsand pyramids. This list is not exhaustive and other types, or non-types,of diagrams are possible. For example, the different elements of thehand-drawn diagrams are individually recognized together with anyspatial and context relationships there between without regard to thediagram type. As discussed earlier, these diagram elements include shapeand text elements. Shape or drawing elements are those that definegraphic or geometric formations in linear or non-linear configurations,and include containers, connectors and free-form drawings. Text elementsare those that contain text characters and include text blocks andlabels for the text blocks and shape elements. Both text blocks andlabels may contain text of one or more characters, words, sentences orparagraphs provided in one or more vertical lines. Text blocks may becontained by containers (internal text blocks) or may be providedoutside of containers (external text blocks). External text blocks maybe unrelated to containers or other elements of a diagram or may bedirectly related to certain other diagram elements.

Further, the application 112 provided by the present system and methodallows users to hand-draw what they have in mind (freely without beingslowed by the technology) as they would on paper, while benefiting fromthe power of digital tools. Example uses include:

-   -   creation: high-level identification of, and differentiation        between, shapes and text allows shapes to be sketched and text        to be written by users without need to select or use        pre-assigned tools when switching between drawing shapes and        writing text,    -   editing: identification of shape and text features enables the        shapes to be manipulated for the change of connections or shape        type, and the handling of editing gestures,    -   typesetting: when creating diagrams users are provided with        options for immediate recognition feedback.

These and other features of the present system and method and nowdescribed in detail.

Through the detection and differentiation of the input of the differenthandwritten objects of shapes and text, the application 112 directsprocessing of the different object types by the HWR system 114 withsuitable recognition techniques, e.g., the strokes of the detectedshapes are processed using a shape language model and the strokes of thedetected text are processed using a text language model. Thisdifferentiation is used by the present system and method to provide live(that is, substantially real-time) recognition feedback to users as thedifferent objects are hand-drawn. However, since many handwritten shapesand text characters can share common features (e.g., a circle and theletter “o”, an arrowhead and the letter “v”) this guidance also providesusers with the ability to correct wrong differentiation decisions,described in detail later.

Through disambiguation handwritten input containing mixed content oftext and non-text (i.e., shapes) is recognized and converted tobeautified digital ink and typeset ink, either automatically (e.g.,on-the-fly) or on demand. Digital ink is formed by rendering thehandwritten input in digital image format. Beautified (digital) ink isformed by rendering the digital ink to appear more regular andnormalized than the original handwriting while retaining similar stylingor look-and-feel. Typeset ink is formed by converting the digital inkinto typeset or fontified image format. Beautified typeset ink is formedby rendering the typeset ink with positional and styling changes fromthe input. The preprocessing stage 116 of the HWR system 114 isconfigured to perform the disambiguation process. The preprocessor 116does this by classifying the elements of the digital ink into differentclasses or categories, being non-text (i.e., shape), text and a mixtureof shape and text. This is done by using simple heuristics such as, forexample, thresholds, writing speed, and time between strokes, or(alternatively or additionally) more complex heuristics such as, forexample, grouping strokes for comparison to spatial and temporalthresholds and language model information, as described in EuropeanPatent Application No. 15290271.4 titled “System and Method ofHandwriting Recognition in Diagrams” filed in the name of the presentApplicant and Assignee and in United States patent application titled“System and Method of Handwriting Recognition in Diagrams” filed in thename of the present Applicant and Assignee contemporaneously with thepresent application, the entire contents of which are incorporated byreference herein. The disambiguation system and method can be performedon already input handwriting or incrementally as handwriting is input.The classified digital ink is then parsed to the recognizer 118 forsuitable recognition processing depending on the classification.

For example, when processing digital ink classified as text, therecognizer 118 employs the segmentation expert 122 to segment individualstrokes of the text to determine the segmentation graphs, therecognition expert 124 to assign probabilities to the graph nodes usingthe classifier 128, and the language expert 126 to find the best paththrough the graphs using, for example, a text-based lexicon of thelinguistic information 130. On the other hand, when processing digitalink classified as non-text, the recognizer 118 employs the segmentationexpert 122 to segment the strokes of the shape, the recognition expert124 to determine segmentation graphs using the classifier 128, and thelanguage expert 126 to find the best path through the graphs using ashape-based lexicon of the linguistic information 130. The mixed contentclassification is treated as ‘junk’ and will result in low probabilityof recognition when parsed to the recognizer 118. Shapes that are parsedto the recognizer and not recognized because, for example, they areout-of-lexicon shapes, are treated as doodles, being unrecognizedcontent (described later).

In editing, handwritten operations such as overwrite, erasure and layoutcontrol, can be performed on both digital and typeset ink of non-textand text. For example, overwriting includes changing a shape from onetype or form to another (e.g., switching a rectangle to an ellipse byhand-drawing an ellipse over a rectangle), adding decoration to aconnector, and creating a shape around text. Erasure can be performedusing known handwriting gestures, such as scratch-out or strike-outgestures on the shapes and text. Layout control can be performed to moveand resize shapes, align and distribute shapes and text to each other orone another. The detection of at least some of these operations, as wellas others, is enhanced by the disambiguation process.

With this classified hand-drawn input, the present system and methodprovides a mechanism for guiding users handwritten input, to optimizerecognition and digital diagram creation. This mechanism can be furtherused to provide real-time recognition feedback to users, which does notdistract from, but enhances, the input of handwriting. This is achievedby providing a guide element which includes a dynamic prompter. Theprompter is used to dynamically display object class recognition, e.g.,non-text or text, and, for shapes, an indication of the recognized shapeand, for text, a certain number of last characters/symbols/words ofrecognized text as typeset ink. The guide element may further include adynamic ruler configured to provide horizontal delimitation forhandwriting and to dynamically grow as strokes are handwritten and/ortypeset ink is rendered in the prompter. The guide element of thepresent system and method is now described with respect to an examplehand-drawn diagram illustrated in FIGS. 4 to 31.

FIG. 4 shows an example hand-drawn diagram 400 input via the UI 402 ofthe application 112 at the user interface 104 of the computing device100, for example. The depiction of the UI 402, and any of itscomponents, is merely an example, and other forms are possible. Thediagram 400 is input including several diagram elements displayed by theapplication 112 as digital ink objects including a container 404containing text 406, a container 408 containing text 410, a connector412 connecting the containers 404 and 408, and a container 414containing text 416. The content and element types are detected andrecognized as described earlier, such that the application 112 handlesthe diagram elements based on their type, e.g., text, non-text and junk(or unrecognized non-text). A guide element 4000 is shown for thecontainer 408 and contained text 410 displayed above and on thecontainer 408 with slight overlap. This rendering and positioning of theguide element is merely an example, and different rendering and/orpositioning is possible, with the constraints that display with minimaldistracting appearance and effect on free handwriting entry in vicinityof the diagram elements(s) to which the guide element pertains ispreferred, as described below.

The guide element 4000 is displayed with a prompter having guidecomponents applicable to the detected and recognized content of thediagram element(s) to which it pertains and control components. Forexample, in FIG. 4, the guide element 4000 has a prompter 4002 includinga shape component 4004 and a text component 4006, and control component4008.

The shape component 4004 is displayed as an icon or label depicting arepresentation of the shape of the container 404 (e.g., a rectangle)recognized by the HWR system 114. However, a generic shape icon can alsobe displayed. In this way, without the need to convert the shape part orall of the digital ink diagram into typeset ink, such as through menu orgesture selection, users are provided with recognition feedback whichinforms them that the hand-drawn shape has been detected, and the typeof shape has been recognized. Accordingly, the users are able tocontinue creation or editing of the diagram in digital ink withoutperforming typesetting until desired.

The text component 4006 is displayed as a single horizontal line oftypeset ink text recognized from the handwritten text 410 by the HWRsystem 114. In this way, without the need to convert the text part orall of the digital ink diagram into typeset ink, users are provided withrecognition feedback which informs them that the handwritten text hasbeen detected, and the content of the text has been recognized.

The control component 4008 is displayed as an interactive controlelement or “more” icon which provides access to control features of theapplication 112. In this way control elements are accessible much closerto the content than with the usual mechanisms, such as the top bar ofthe application.

In order to position the guide element close to the diagram elements towhich it pertains, the present system and method may be made withrespect to an alignment grid underlying the display interface, whichgrid may also be used to define one or more extents and positioning ofthe diagram elements themselves. FIGS. 5 to 28 show different displayformats of the guide element 4000 in relation to examples of a sequenceof handwritten input of the shapes and text of the diagram 400 andinteractions performed on the guide element 4000.

FIG. 5 shows input of the text element 406 as a first hand-drawn elementof the diagram 400. Upon detection of the beginning of input and therendering of the corresponding digital ink on the interface 104 thepresent system and method initiates display of the guide element 4000 asa text prompter 4010 which has a horizontal line or bar 4012 contiguouswith and above the handwritten characters of the text 406 (as set by thealignment grid, for example), and the text component 4006 includingtypeset ink text 4014 recognized from the text 406 above the horizontalbar 4012, i.e., the word “Ok”. The horizontal bar 4012 is configured todynamically grow as text characters are input and recognized content isdisplayed in the text prompter 4010. This growth is limited however, andeffects such as automatically scrolling to the left of the text prompterof already displayed recognized text may be made to ensure newlyrecognized content is displayed in the text prompter 4010.

Accordingly, substantially real-time content recognition feedback isprovided during text input by the text prompter 4010. The controlelement 4008 is also displayed in the text prompter 4010 to the right ofthe text component 4006 and above the horizontal bar 4012. However, inorder to not distract users during content input, the control icon mayonly be displayed when input is not being performed, e.g., duringwriting the ‘more’ icon is kept invisible and after writing, that is, atdetection of ceasing of an input event, e.g., finger/pen up event, plusa predefined time period (e.g., about half a second to about a second)the ‘more’ icon is faded into full visibility.

Further, to not distract user input, the text prompter 4010 (and the‘more’ icon 4008) is displayed with light and subtle rendering, e.g., inlight blue or at least a different shade of the color used for thediagram element digital and/or typeset ink display, that is just betweenvisible and faded. In this way the guide element is noticeable butunobtrusive so as to guide the handwriting input without distractingfrom the diagram creation itself. Further, the display of the guideelement remains until different input is received by the application112, or the end of a predefined time period (e.g., about half a secondto about a second) after the input cessation event and during which nofurther input to the element the guide element pertains is detected,with fading of the guide element into invisibility.

FIG. 6 shows the subsequent input of the shape element 404 drawn as acontainer containing the already input text 406. At this point, theshape element 404 may be identified as a container which contains therecognized word 406 due to the relative positions and characteristics ofthe inputs 404 and 406. A container and text contained thereby areassociated with one another, so that some actions performed on one ofthese elements causes reactions on the other element. For example, whenthe container is selected and moved by the user the contained text ismoved with the container, and when the text is selected and enlarged oradded to by the user, the container is resized to accommodate the largertext size or block.

Upon detection of the beginning of input and the rendering of thecorresponding digital ink on the interface 104 the present system andmethod, for example, omits display of the text prompter 4010 (along withthe control element 4008) and initiates display of the guide element4000 as a shape prompter 4016 above the hand-drawn shape 404 (as set bythe alignment grid, for example) which has the shape component 4002including a circle graphic 4018 recognized from the shape 404.Accordingly, substantially real-time content recognition feedback isprovided at shape input. So as not to distract user input, the shapeprompter 4016 is displayed with the same or similar light and subtlerendering as the text prompter 4010 and display is retained in a similarmanner.

FIG. 7 shows the container 404 and contained text 406 of FIG. 6 but withthe shape prompter 4016 omitted from display and the text prompter 4010re-displayed so as to partially overlay the circle 404. This re-displayis performed due to detection of a selection gesture on the digital inktext 406, such as a single-point gesture, e.g., a single tap, or amulti-point gesture, e.g., a double tap, by the application 112.Accordingly, users are able to interact with the hand-drawn content toascertain the recognition status of the content, for example. Thedisplay of the guide element for selected content remains until newinput, such as handwriting or a gesture, is received by the application112, or for a predefined time period during which no interaction withthe guide element or the content to which it pertains (e.g., about fiveseconds) is detected, with fading of the guide element intoinvisibility.

FIG. 8 shows the subsequent input of a portion 412 a of the shapeelement 412 drawn as a bent line below the circle 404. The shape element412 is an arrow, as seen in FIG. 4, which is defined as being formed bya straight or bent ‘stem’ terminating in one or two open- (e.g.,v-shaped) or closed- (e.g., triangular) ‘arrowheads’. The portion 412 aconstitutes the stem of the arrow 412. Upon detection of the beginningof input and the rendering of the corresponding digital ink on theinterface 104 the present system and method initiates display of theguide element 4000 as the shape prompter 4016 beside the hand-drawnshape 412 a (as set by the alignment grid, for example) which has theshape component 4002 including a line graphic 4020 recognized from theshape 412 a, i.e., a line. It is noted that the line graphic shown inFIG. 8 is a generic line; however the graphic may be a bent line in thisexample like the recognized bent line 412 a.

At this point, the line 412 a may be identified as a connector of thecircle 404 due to their relative positions (e.g., a pre-set spatialseparation threshold is used by the application 112, where separation ofan end of a linear shape to the non-linear shape below that thresholdindicates a high likelihood of a connection relationship; the spatialthreshold may be defined as a distance in pixels between mean points orbarycenters of the strokes, for example, set to be about five pixels toabout 100 pixels) and/or characteristics of the inputs 404 and 412 a(e.g., a line having one end proximate or adjoining a containerindicates a high likelihood of a connection relationship; with proximitydefined in a distance range similar to the spatial threshold). A shapeobject, such as a container, and a connector connected thereto areassociated with one another so that some actions performed on one ofthese elements causes reactions on the other element. For example, whenthe container is selected and moved by the user the connector is movedwith the container.

FIG. 9 shows the subsequent input of a portion 412 b of the shapeelement 412 drawn as a rotated v-shape at the end of the line 412 awhich is not proximate the circle 404. The portion 412 b constitutes thearrowhead of the arrow 412. Upon detection and recognition of theportion 412 b as the arrowhead of the arrow 412 made up of the bent line412 a and the arrowhead 412 b, the recognized shape graphic of the shapeprompter 4016 is changed to an arrow graphic 4022, without repositioningof the shape prompter 4016, however repositioning is possible. It isnoted that the graphic shown in FIG. 9 is a generic arrow; however thegraphic may be a bent arrow in this example like the recognized bentarrow 412.

The determination of the later drawn shape 412 b and the earlier drawnshape 412 a as belonging to the single shape 412 is made by theapplication 112 based on a probability score. That is, the strokes 412 aand 412 b of the arrow 412 are drawn within a relatively long space oftime, say, more than one second (e.g., greater than a pre-set, andre-settable, temporal separation threshold used by the application 112,where the time separation between drawing strokes below that thresholdindicates a high likelihood of the strokes being parts of a singleshape), such that the stroke 412 a is parsed to the recognizer 118 aloneand recognized as a bent line, as described with respect to FIG. 9. Theproper identification of the complete connector arrow 412 is providedhowever through the overall probability score calculated from spatialand temporal scores and metrics or characteristics of the input strokes.That is, while the temporal score for the strokes 412 a and 412 b beingpart of the same object is low, the combination of the high spatialscore for the strokes 412 a and 412 b being part of the same object, asreturned through the grouping performed by the disambiguation system andmethod and the characteristics (e.g., a rotated v-shape at the end of aline) result in the overall probability of the strokes 412 a and 412 bbelonging to one shape being high. This determination of an arrowheadadded to the line 412 a can be used to affirm the connector status ofthe shape element 412, if arrows are classed by the application 112 ashaving high likelihood of being connectors, for example.

FIG. 10 shows an alternative input of a portion 412 b′ of the shapeelement 412 drawn as a rotated v-shape like the portion 412 b shown inFIG. 9 but drawn at a distance from the end of the line 412 a unlike theportion 412 b shown in FIG. 9. At the rendering of the digital ink,display of the guide element 4000 as the text prompter 4010 is made withthe text component 4006 including typeset ink text 4024 recognized fromthe portion 412 b′, i.e., the symbol “>”. The portion 412 b′ isrecognized as text instead of as part of the shape 412, due to therelatively large distance between the portion 412 a and the portion 412b′ resulting in a higher probability score that the input 412 b′ belongsto a separate object, and therefore the disambiguation system and methodis able to classify the portion 412 b′ as text, instead of non-text. Thepresent system and method however provides users with the ability tochange this recognition result, as follows.

Each of the shape, text and control components of the guide element 4000are interactive in either the concurrent display (i.e., as in FIG. 10)or post-display (i.e., as in FIG. 7) modes of the guide element 4000.This interactivity can be accessed through selection with the detectionof a selection gesture on a component, such as a single-point gesture,e.g., a single tap or long press, by the application 112, resulting inassigned actions for that component with respect to the display mode ofthe guide element, e.g., concurrent or post. Accordingly, users are ableto interact with the hand-drawn content to ascertain the recognitionstatus of the content, for example. FIG. 11 shows the input stage ofFIG. 10 with a menu 4026 displayed in response to detection of aselection gesture on the recognized text 4024 in the text prompter 4010.The text menu 4026 has two display zones 4026 a and 4026 b.

In the menu zone 4026 a, a list of text recognition candidates isdisplayed. These candidates are the most likely recognition candidatesoutput by the HWR system 114 with respect to the recognition process forthe portion 412 b′. In FIG. 11, the four best candidates are shown,however more or less than these can be displayed. Each of the displayedcandidates in the zone 4026 a are individually interactive throughselection with the detection of a selection gesture on a candidate, suchas a single-point gesture, e.g., a single tap or long press, by theapplication 112, resulting in the recognized text 4024 in the textprompter 4010 being changed to the selected candidate, and the selectedcandidate to be displayed when the portion 412 b′ is typeset. In themenu zone 4026 b, a manual classification action is displayed, which forthe text menu is “Text to shape”. This action of the zone 4026 b issimilarly interactive, with selection causing the recognized text 412 b′to be re-classified as non-text with consequential re-recognition beingperformed. In this process, the disambiguation system and methodre-evaluates the now classified shape 412 b′ with other alreadyclassified proximate shape elements, i.e., the line 412 a, to determineif groupings of the shape 412 b′ with those other shape elements resultin a probability of the shape 412 b′ being part of a greater shapeelement, i.e., the shape element 412, for consequential recognition bythe recognizer 118. The displayed actions are only an example, and more,less or different actions may be displayed.

FIG. 12 shows the input stage of FIG. 10 with a different menu 4028displayed in response to detection of a selection gesture on the ‘more’icon 4008 in the text prompter 4010. The text control menu 4028 has twodisplay zones 4028 a and 4028 b. In the menu zone 4028 a, actionsrelated to the content transform are displayed, which for the textcontrol menu is the “Text to shape” action, a “Reflow” action and a“Typeset” action. These actions of the zone 4028 b are individuallyinteractive, with selection of the “Text to shape” action causing manualre-classification as described above, of the “Reflow” action causing,for the text control menu 4028, reflow of the digital or typeset inktext within or onto multiple lines, and of the “Typeset” action causing,for the text control menu 4028, typesetting of the displayed digital inktext. In the menu zone 4028 b, interactive actions related to standardoperations are displayed, which for the text control menu are cut, copy,delete, export and properties/settings. These displayed actions are onlyan example, and more, less or different (with the exception of manualreclassification) actions may be displayed.

FIG. 13 shows the connector 412 of FIG. 6 but with the shape prompter4016 omitted from display and a connector prompter 4030 of the guideelement 4000 in proximity of the connector 412 (as set by the alignmentgrid, for example). The connector prompter is displayed due to detectionof a selection gesture on the connector 412, such as a single-pointgesture, e.g., a single tap or long press, by the application 112. Theselected connector 412 is displayed in selection mode rendering whichincludes the typeset version of the shape overlaid on the digital ink,with the rendering of the digital ink de-emphasized (e.g., displayedlighter) and the typeset ink emphasized (e.g., displayed in a colorand/or ink weight greater than that of the normal ink display), andselection handles A, for resizing of the connector. The connectorprompter 4030 has a connector component 4032 including the arrow graphic4022 recognized from the shape 412. The control element 4008 is alsodisplayed in the connector prompter 4030 to the right of the connectorcomponent 4032 and above a horizontal bar 4012.

As the connector prompter is displayed in response to user interactionwith a connector, the connector prompter 4030 is displayed with bolderrendering than the shape and text prompters, since interaction with theconnector prompter is likely expected. The display of the connectorprompter remains until de-selection of the selected connector isdetected, or the end of a predefined time period during which nointeraction with the connector prompter or the content to which itpertains (e.g., about five seconds) is detected, or different input isreceived by the application 112, with fading of the guide element intoinvisibility.

As with the text prompter, the connector prompter is interactive. FIG.14 shows the input stage of FIG. 13 with a menu 4034 displayed inresponse to detection of a selection gesture 4036 on the connector icon4032 in the connector prompter 4030. The connector menu 4034 has twodisplay zones 4034 a and 4034 b. In the menu zone 4034 a, a list ofalternative connector arrow types are displayed which are individuallyinteractive. These alternatives are displayed based on the type ofconnector being interacted with, in this example an arrow. For the lineconnector of FIG. 8 a different list of alternative connector types areshown in the connector menu. In the menu zone 4034 b, actions which areindividually interactive and related to the content transform aredisplayed, which for the connector menu include a “Set as text” actionand a “Set as doodle” action, with selection of the “Set as text” actioncausing manual re-classification as described above, and of the “Set asdoodle” action causing the recognition of the shape 412 as a connectorto be changed to doodle, thereby removing the connector status of thereclassified doodle with the above-described consequential effect on thedigital ink rendering even after typesetting of the diagram 400. Thesedisplayed actions are only an example, and more, less or differentactions may be displayed.

FIG. 15 shows the input stage of FIG. 13 with a menu 4036 displayed inresponse to detection of a selection gesture on the ‘more’ icon 4008 inthe connector prompter 4030. The connector menu 4036 has two displayzones 4036 a and 4036 b. In the menu zone 4036 a, the interactiveactions related to standard operations discussed above are displayed. Inthe menu zone 4036 b, the “Typeset” content transform action isdisplayed. These displayed actions are only an example, and more, lessor different actions may be displayed.

FIG. 16 shows the diagram 400 after subsequent input of the container408 and contained text 410 and with the subsequent input of the textelement 416. Upon detection of the beginning of input and the renderingof the corresponding digital ink on the interface 104 the present systemand method initiates display of the guide element 4000 as the textprompter 4010 having the text component 4006 including the typeset inkword 4014 recognized from the text 416 above the horizontal bar 4012,i.e., the word “Operation”.

FIG. 17 shows an alternative input of the text element 416 having aportion 416 a, the letter “O”, and a portion 416 b, the letters“peration” which are written at a distance from one another. At therendering of the digital ink, display of the guide element 4000 as boththe shape prompter 4016 above the portion 416 a and the text prompter4010 above the portion 416 b is made. This is because the portion 416 ais recognized as a shape, namely a circle with display of the circlegraphic 4018, instead of text, and the portion 412 b′ is properlyrecognized as text. This is due to the relatively large distance betweenthe portions 416 a and 416 b resulting in a higher probability scorethat the inputs 412 b and 416 b belong to separate objects, andtherefore the disambiguation system and method is able to classify theportion 416 a as non-text, instead of text. As before, this recognitionresult can be changed through interaction with the shape prompter 4016to launch a menu.

FIG. 18 shows the input stage of FIG. 17 with a menu 4038 displayed inresponse to detection of a selection gesture on the shape component 4004in the shape prompter 4016. The shape menu 4038 has two display zones4038 a and 4038 b. In the menu zone 4038 a, a list of shape recognitioncandidates are displayed. These candidates are the most likelyrecognition candidates output by the HWR system 114 with respect to therecognition process for the portion 416 a. In FIG. 18, the four bestcandidates are shown, however more or less than these can be displayed.Each of the displayed candidates in the zone 4038 a are individuallyinteractive through selection with the detection of a selection gestureon a candidate, such as a single-point gesture, e.g., a single tap orlong press, by the application 112, resulting in the shape graphic 4018in the shape prompter 4016 being changed to the selected candidate, andthe selected candidate to be displayed when the portion 416 a istypeset. In the menu zone 4038 b, a manual classification action isdisplayed, which for the shape menu is “Shape to text”. This action ofthe zone 4038 b is similarly interactive, with selection causing therecognized shape 416 a to be re-classified as text with consequentialre-recognition being performed. In this process, the disambiguationsystem and method re-evaluates the now classified text 416 a with otheralready classified proximate text elements, i.e., the text 416 b, todetermine if groupings of the text 416 a with those other text elementsresult in a probability of the text 416 a being part of a greater textelement, i.e., the text element 416, for consequential recognition bythe recognizer 118. The displayed actions are only an example, and more,less or different actions may be displayed.

FIG. 19 shows the subsequent input of the shape element 414 (from theinput of FIG. 16) as the last element of the diagram 400 drawn as acontainer containing the already input text 416. At this point, theshape element 414 may be identified as a container which contains therecognized word 416 due to the relative positions and characteristics ofthe inputs 414 and 416. Upon detection of the beginning of input and therendering of the corresponding digital ink on the interface 104 thepresent system and method, for example, omits display of the textprompter 4010 and initiates display of the guide element 4000 as theshape prompter 4016 above the hand-drawn shape 414 (as set by thealignment grid, for example) which has the shape component 4002including an ellipse graphic 4018 recognized from the shape 414.

FIG. 20 shows the container 414 and contained text 416 of FIG. 19 butwith these elements displayed in the afore-described selection moderendering which includes the selected typeset version of the shape and aselected digital ink version of the text (however the text could berendered in selected typeset as well) due to detection of a selectiongesture on the container 414, for example. In response, the full guideelement 4000 of the shape prompter 4016, the text prompter 4010 andcontrol element 4008 is displayed, so that the entire selected diagramelement of the container and contained text can be interacted with viathe ‘more’ icon, or the shape and text elements can be individuallyinteracted with via the respective prompters.

FIGS. 21A and 21B illustrate alternative scenarios of this selection inwhich the text element 416 and the shape element 414 respectively havebeen input separately and alone. In such a case, the selection of thelone elements causes selection mode display of that element and displayof the corresponding guide element prompter (with the control element).

FIG. 22 shows the completed diagram 400 displayed in typeset ink inresponse to user interaction with the application 112 to performtypesetting, for example. That is, the diagram elements are displayed asa typeset container 404 b containing typeset text 406 b, a typesetcontainer 408 b containing typeset text 410 b, a typeset connector 412 cconnecting the typeset containers 404 b and 408 b, and a typesetcontainer 414 b containing typeset text 416 b.

FIG. 23 shows the typeset diagram 400 with the typeset container 414 band contained text 416 b displayed in the afore-described selection moderendering which includes the selected typeset version of the shape andtext due to detection of a selection gesture on the container 414 b, forexample. In response, the full guide element 4000 of the shape prompter4016, the text prompter 4010 and the control element 4008 is displayed,as before.

FIGS. 24, 25 and 26 respectively show the shape, text and control menus4038, 4026 and 4028 variously having the afore-described interactiveactions displayed upon detection of selection gestures on the shapeprompter 4016, the text prompter 4010 and the control element 4008,respectively.

FIG. 27 shows the typeset diagram 400 with the typeset container 408 band contained text 410 b and the typeset container 414 b and containedtext 416 b displayed in the afore-described selection mode renderingwith a selection box 4040 about all of the selected elements, due todetection of a multi-selection gesture, such as a single-point gestureon one of the containers 408 b and 414 b, e.g., a long press, followedby another single-point gesture on the other containers, e.g., a shorttap, by the application 112. In response, the guide element 4000 isdisplayed including a multi-element prompter 4042 and the controlelement 4008. Like the other prompters and components of the guideelement, the multi-element prompter 4042 is interactive.

FIG. 28 shows the selection mode of FIG. 27 with a menu 4044 displayedin response to detection of a selection gesture 418 on the multi-elementprompter 4042. The multi-element menu 4044 displays a list of manualclassification actions, including the afore-described “Set as text”,“Set as shape” and “set as doodle” actions. Each action of the menu 4044is individually interactive through selection with the detection of aselection gesture thereon, such as a single-point gesture, e.g., asingle tap or long press, by the application 112. Selection of theseactions causes all elements of the multi-selection to be re-classifiedand re-recognized accordingly. In the multi-select mode, selection ofthe control element 4008 launches the control menu 4028 with theinteractive elements described earlier.

As described earlier, input shapes that are parsed to the recognizer butnot recognized because, for example, they are out-of-lexicon shapes, aretreated as doodles, and therefore not rendered in typeset ink in atypesetted diagram. This non-recognition occurs where the recognizer 118is configured to only return recognition output if a candidate above acertain probability score threshold is determined. An example doodleinput to the diagram 400 is now described with reference to FIGS. 29 to31.

FIG. 29 shows the input of a shape element 420 drawn as a free-formshape above the typeset container 408 b. The new input to the typesetdiagram 400 is displayed as digital ink, unless a setting has been madeto incrementally typeset new input. At this point, the shape element 404is identified as non-text by the disambiguation system and method but isnot recognized by the recognizer 118 as it is an out-of-lexicon shape.In order to provide users with feedback of this failed recognition, upondetection of the beginning of input and the rendering of thecorresponding digital ink on the interface 104 the present system andmethod initiates display of the guide element 4000 as a free-formprompter 4046 above the hand-drawn shape 420 (as set by the alignmentgrid, for example) which has a free-form component 4048 including adoodle graphic 4050. So as not to distract user input, the free-formprompter 4046 is displayed with the same or similar light and subtlerendering as the other prompters and display is retained in a similarmanner.

FIG. 30 shows the typeset diagram 400 with the digital ink shape 420displayed in the afore-described selection mode rendering with aselection box 4040 about the selected element, due to detection of aselection gesture, such as a single-point gesture, e.g., a short tap, onthe shape by the application 112. In response, the guide element 4000 isdisplayed including the free-form prompter 4046 and the control element4008. Like the other prompters and components of the guide element, thefree-form prompter 4046 is interactive.

FIG. 31 shows the selection mode of FIG. 30 with a menu 4052 displayedin response to detection of a selection gesture 422 on the free-formcomponent 4048. The doodle menu 4052 displays a list of manualclassification actions, including the afore-described “Set as text” and“Set as shape” actions. Each action of the menu 4052 is individuallyinteractive through selection with the detection of a selection gesturethereon, such as a single-point gesture, e.g., a single tap or longpress, by the application 112. Selection of these actions causes theselected non-recognized free-form element to be re-classified andre-recognized (for example, this manual reclassification may prompt therecognizer 118 to return the best candidate for the free-form shapeunder the probability score threshold) accordingly.

While the foregoing has described what is considered to be the best modeand/or other examples, it is understood that various modifications maybe made therein and that the subject matter disclosed herein may beimplemented in various forms and examples, and that they may be appliedin numerous other applications, combinations, and environments, onlysome of which have been described herein. Those of ordinary skill inthat art will recognize that the disclosed aspects may be altered oramended without departing from the true spirit and scope of the subjectmatter. Therefore, the subject matter is not limited to the specificdetails, exhibits, and illustrated examples in this description. It isintended to protect any and all modifications and variations that fallwithin the true scope of the advantageous concepts disclosed herein.

We claim:
 1. A system for guiding hand-drawing of diagrams includingtext and non-text elements on a computing device, the computing devicecomprising a processor and at least one non-transitory computer readablemedium for detecting and recognizing hand-drawing diagram element inputunder control of the processor, the at least one non-transitory computerreadable medium configured to: accept input of, on an input area of aninteractive display of the computing device: a non-text hand-drawingdiagram element, and a selection gesture; and cause display of, on theinteractive display: a digital ink of the non-text hand-drawing diagramelement, a recognized form of the non-text hand-drawing diagram elementoverlaid on the digital ink of the non-text hand-drawing diagramelement, wherein the recognized form of the non-text hand-drawingdiagram element includes one or more geometric characteristics of thedigital ink of the non-text hand-drawing diagram element and permitsmodification of one or more of the geometric characteristics of therecognized form of the non-text hand-drawing diagram element, and adynamic prompter which dynamically displays at least one depiction ofthe non-text hand-drawing diagram element, the dynamic prompterconfigured with the recognized form of the non-text hand-drawing diagramelement and geometrically similar variants of the non-text hand-drawingdiagram element; wherein the computer readable medium is configured tocause display of the dynamic prompter in response to the selectiongesture selecting the digital ink of the non-text hand-drawing diagramelement; and wherein the recognized form of the non-text hand-drawingdiagram element is initially determined based on a probability scorecalculated from at least one of: a spatial score, a temporal score, andan input characteristic of the non-text hand-drawing diagram element. 2.A system as claimed in claim 1, wherein the dynamic prompter dynamicallydisplays at least one depiction of a handwriting text diagram element ashandwriting is input.
 3. A system as claimed in claim 1, wherein thedynamic prompter includes at least one interactive prompter element, theat least one non-transitory computer readable medium configured todisplay an action menu in response to receiving interaction with the atleast one interactive prompter element.
 4. A system as claimed in claim1, the at least one non-transitory computer readable medium configuredto display the depiction of a non-text element in icon form.
 5. A systemas claimed in claim 4, wherein the non-text element is a shape elementof the diagram, the icon depicting the shape.
 6. A system as claimed inclaim 1, the at least one non-transitory computer readable mediumconfigured to display the depiction of a text element in typeset ink. 7.A method for guiding hand-drawing of diagrams including text andnon-text elements on a computing device, the computing device comprisinga processor and at least one non-transitory computer readable medium fordetecting and recognizing hand-drawing diagram element input undercontrol of the processor, the method comprising: inputting, on an inputarea of an interactive display of the computing device: a non-texthand-drawing diagram element, and a selection gesture; and displaying,on the interactive display: a recognized form of the non-texthand-drawing diagram element overlaid on the digital ink of the non-texthand-drawing diagram element, wherein the recognized form of thenon-text hand-drawing diagram element includes one or more geometriccharacteristics of the digital ink of the non-text hand-drawing diagramelement and permits modification of one or more of the geometriccharacteristics of the recognized form of the non-text hand-drawingdiagram element, and a dynamic prompter which dynamically displays atleast one depiction of the non-text hand-drawing diagram element, thedynamic prompter configured with the recognized form of the non-texthand-drawing diagram element and geometrically similar variants of thenon-text hand-drawing diagram element; wherein the dynamic prompter isdisplayed in response to selecting, via the selection gesture, thedigital ink of the non-text hand-drawing diagram element; and whereinthe recognized form of the non-text hand-drawing diagram element isinitially determined based on a probability score calculated from atleast one of: a spatial score, a temporal score, and an inputcharacteristic of the non-text hand-drawing diagram element.
 8. A methodas claimed in claim 7, wherein the dynamic prompter dynamically displaysat least one depiction of a handwriting text diagram element as thehandwriting is input.
 9. A method as claimed in claim 7, wherein thedynamic prompter includes at least one interactive prompter element, theat least one non-transitory computer readable medium configured todisplay an action menu in response to receiving interaction with the atleast one interactive prompter element.
 10. A method as claimed in claim7, comprising displaying the depiction of a non-text element in iconform.
 11. A method as claimed in claim 10, wherein the non-text elementis a shape element of the diagram, the icon depicting the shape.
 12. Amethod as claimed in claim 7, comprising displaying the depiction of atext element in typeset ink.
 13. A non-transitory computer readablemedium having a computer readable program code embodied therein, saidcomputer readable program code adapted to be executed to implement amethod for guiding hand-drawing of diagrams including text and non-textelements on a computing device, the computing device comprising aprocessor and at least one non-transitory computer readable medium fordetecting and recognizing hand-drawing diagram element input undercontrol of the processor, the method comprising: inputting, on an inputarea of an interactive display of the computing device: a non-texthand-drawing diagram element, and a selection gesture; and displaying,on the interactive display: a recognized form of the non-texthand-drawing diagram element overlaid on the digital ink of the non-texthand-drawing diagram element, wherein the recognized form of thenon-text hand-drawing diagram element includes one or more geometriccharacteristics of the digital ink of the non-text hand-drawing diagramelement and permits modification of one or more of the geometriccharacteristics of the recognized form of the non-text hand-drawingdiagram element, and a dynamic prompter which dynamically displays atleast one depiction of the non-text hand-drawing diagram element, thedynamic prompter configured with the recognized form of the non-texthand-drawing diagram element and geometrically similar variants of thenon-text hand-drawing diagram element; wherein the dynamic prompter isdisplayed in response to selecting, via the selection gesture, thedigital ink of the non-text hand-drawing diagram element; and whereinthe recognized form of the non-text hand-drawing diagram element isinitially determined based on a probability score calculated from atleast one of: a spatial score, a temporal score, and an inputcharacteristic of the non-text hand-drawing diagram element.
 14. Anon-transitory computer readable medium as claimed in claim 13, whereinthe dynamic prompter dynamically displays at least one diagram elementas the handwriting is input.
 15. A non-transitory computer readablemedium as claimed in claim 13, wherein the dynamic prompter includes atleast one interactive prompter element, the at least one non-transitorycomputer readable medium configured to display an action menu inresponse to receiving interaction with the at least one interactiveprompter element.
 16. A non-transitory computer readable medium asclaimed in claim 13, the method comprising displaying the depiction of anon-text element in icon form.
 17. A non-transitory computer readablemedium as claimed in claim 16, wherein the non-text element is a shapeelement of the diagram, the icon depicting the shape.
 18. Anon-transitory computer readable medium as claimed in claim 13, themethod comprising displaying the depiction of a text element in typesetink.
 19. A system as claimed in claim 1, wherein the recognized form ofthe non-text hand-drawing diagram element is visually emphasizedrelative to the digital ink of the non-text hand-drawing diagramelement.
 20. A system as claimed in claim 19, wherein the visualemphasis consists of bolding the recognized form of the non-text handdrawing diagram relative to the digital ink of the non-text hand-drawingdiagram element.